3D Motion Estimation using a Token Tracker

111 1.11is u t i c l e we descr ibe the i ~ ~ ~ p l e t ~ ~ e ~ ~ t a t i u ~ ~ o f a sys tem l e v 1 1r.11 k i n g t c ~ k e l ~ s o n a sequence o f perspective v iews a n d use IIIV res111t t o c o n ~ p u t e the YL) ~ n o t i o ~ ~ f a cantera a n d t h e 3D -1 I 11, I u r r ul' the scene ( u p l o a scirle factur) . Wt. ~ w o v ~ d e t i rs t a l l overv iew u f the ent i re process fo r mea;III IIIK IIII~I~I: flow. We 111e11 p r e s e l ~ t a paranretr ic r c p r e s e ~ l t a t i o n I.,, IIIIV SV~I I I~ I ILJ wh ich s i ~ ~ ~ p l i t i e s the i ~ ~ i l ~ l e ~ ~ ~ e t ~ t a t i ~ ~ ~ o f t rack1111: vtlge l i ~ l r s . l'l~i~ is fo l luwed by tile J e s c r l l ~ t . i o ~ ~ o f predict ion, III.III II~IIC: .IIISI 11pd.11i11g o f t,Itc iill.ige I l t ~ w 111uelel. T h i s f o r ~ ~ r a l i ~ ~ r III,IL~.. IIO~~II~IC. a fast 111atc11i11g ~ ~ ~ ( I ~ I I ~ I I I I . 1'111.tlly we r u ~ ~ s i d e r t h e 3D 111utiu11 e s t u ~ ~ a t i u i ~ si11g t h e tz1 1 3 r~,r/rl:s o f ~ h e ~ e y r n e n t s he ing t racked 'I'l~e fact thd t they d u n o t Q ,,I I ~ , ? I N B I I ~ ~ ill g w e r a l t o the same p l~ys ic ill poi111, i~ a m a j o r source I t I AIIIILII~.~ o11e is the preseuce u f zncorrecl mulches. 111 t h i s t . & . a IIW ~ A I ~ I I I I ~ tea I l l l iques for IIIOIIUII e>tit11~1Lin11 f a i l w l ~ i l e 011r I,.. I~IIII~IIV I ) I O V ~ ~ p a r ~ i c u l a r l y r o b u ~ t : t l ~ e i l l co r rec t ~ l ~ a t c l r e s are a1~1c.a IV+I Ily tl118 t l ~ o t i u n e s t i n l a t i o ~ ~ 1111it and the i l r fo r l~ l i r t i o l l on 1.11. t ~ ~ ~ I I ~ ~ ~ ~ J I ~ I I ~ ~ ~ ~ I I ~ ~ J are given Lo the tukr11 t racker. I, 1 IIII,.IILJ 11,1ve been car r ied ~III~ O tc.11 i t l ~ a g e sque l l ces I .eLa,l, l q .I I I I ~ ~ V I I I ~ I .ullt:ra, s l ~ u w i n g tl1,11 31) IIIUI io11 a11c1 YD strucI ~ I I ,. ,V.IIIII.~I~~~II is 1.1s~ a n d r e l i a l l e . I 'I'IIE TOKEN TRACKER 1 ;IV<.II .r d r q u u w e o f ilnages, one has 11, tr;rck t ~ ~ o v i l r g u l ~ j e c t s in I 11,. st c.11t.. O u r approach uses tokens based UII l l~oe segr l~ rn ta cor~ t ~ . ~ ~ o l l ~ l i l l g t o 111e edges ex t rac ted f r o m the sraue. However, it is u . p r ~ l r w l ~ i l v l a , uo le t l ~ a t o ther t o k ~ l l d as pn iu ts u f in terest (COTII-.IS, tvi l t le poit l ts..) ca l l be co~~s iderec l w i t h o l ~ t aHert i l rg deeply 111,. r ~ l j i t ~ ~ ~ i l l ~ ~ ~ ~ . 'l'11e edges are 01~tai11c:J tI1roug11 the use o f a n I.III IIII~II J ) C ~ ~ ~ U I . previousIy d e v e l c q ~ e ~ l 11 1. AII edge l i l ~ k u ~ g s tep . I I~~I , f i IUDI~~UII~I a p l ~ r o x i t t l a t i o ~ ~ give the Iit le seg111e11ta 011 wh ich I 11,. I I .I# L I I I ~ is (IoII~. O u r track111g approat 11 is I>asrtI 011 a comI,III~ILII,II UI il pred ic t ion step JII~ a I I I ~ L ~ ~ I ~ I I ~ ~ ~ r o ~ e s s . K a l l n a n II~IVIIIIK la 11seJ t o 11elp t r a r k i l l g I>y l , ruv lc l i~~g rc 'wonable estiIII.~IV* #>I' the r e g i o ~ ~ where t h e n1~rtc11111g yrocess 11.1s t o seek f o r .I t~c~;.;.il,la 111atc11 bctween t o k e l ~ s . C u r r e s l r u ~ l c l e ~ ~ r e ill tlre search .,IV.I 1.1 ~ I ~ ~ I I ~ t11rougl1 the use o f a s i ~ ~ l i l i r r i t y f u ~ ~ r t i o n based o n .-I lam): 1r.11 l ~ r e s 01 (Ire l i ue s e g ~ t ~ e n t s . W11t.11 wurk ing w i t h a large >4'11111'111 t. 111 fr.1111t.s, i t is possible t11.1t sttt l lr olr jects 111ay appear 1 1 1 ,.. I (4,t.tlly v r pa r t ia l l y ; OII K.I~III.~II l i l t r r i ~ ~ g based apI~I~..,,I, . , ~ l ~ , w ~ t u I ~ , u ~ ( l l e t h i s p r o I ) l e ~ ~ l vl G N ~ 111siot1 ill il etfective w ,, ~ ~ . ~ ~ ~ ~ ~ I I I I I ~ ~ I I L Y I IV~ t1ee11 carr ied LIII~ OII 11~1sy s y n t l ~ e t i c d a t a .#IOI a n 1 0 IY.II *(CIICY ~ ~ L t ~ ~ i l l e c l ~I,UIII a IIIUI,IIC IC~IUI. SOIII~ exper i I,,* I,I ,,I I~.SIIII~ 1 OII, t . r ~ l i l l g t l~t: rei11 scet~es .IV a l f i a ~ w ~ ~ I h a 11 I~A,,I, ( i e. l i ~ l e s e U t l r e l ~ ~ ) k ~11; r ra r~er iucd Iny t lre f u l l u w i ~ ~ g I I V ~ IB.II'.LII&~L~~Y . ' 1 ' 1 1 ~ ,)II~II~:I~~UII 0 o f t l ~ e I iue seg111e11t. ' 1 ' 1 1 ~ 111ag11itu~1e u f t11e g r a \ i ~ e ~ l t a lung tlre IIII~ seg111e11t. T h e 1~11gt l1 L o f the l i n e segnlent. T h e d i s t a t ~ c e o f the o r i g i n t o t lre l iue segnrent deno ted b y l l ~ e paral l re ler c. ' r l ~ e J h t a l l c e denoted d , a l u ~ ~ g tile l iue f ro tn the perpendicu1.1r i~ l te rsec t iun t o t h e 111ic1poi11t u f t h e segnrent. A K a l ~ l i a r r f i l ter i used t o a i d t r a r k i l ~ g by p rov id ing reasonable eatitnates o f ~ l r r eg ion wl lere t l ~ e ~ ~ ~ a t c l l i l ~ g process has t o seek fo r a possil,le m a t c h b r t w c e n t u k v l ~ s . K a L n a n f i l t e r ing is ;I s t i ~ t i ~ r i c ,rl a l ~ p r o ~ c h t o es t imate a t imeva ry ing state vec to r XI fi.*)tl~ l loidy III~UII~CIII~II~S Z1. Consider t h e es t imat ion o f Xttr I~OIII 111s II~..L III.(.IIIC.IILY u p t u t h e ins tan t t, Ka lma l r f i l t e r ing i s a tecur.ivc c j t i l ~ i ~ l l i ~ r ~ ~ schen e desig~red t o m a t c h tlre d y n a m i c syste ln 111t~cle1, t l ~ c st; l t ist ics o f t l re e r ro r between the m o d e l ancl real i ty , a11J t llr IIIIC~I t a i n t y associated w i t h t h e measurements. I11 o u r approach, a K a111la11 li11t:r is used separate ly o n each o f t h e f ive parameters d e t i ~ ~ e t l al,ovr t o c s t i ~ l l a t e a state vector w h i c h is s imp ly the t in le va ry i l l g 111tdiu11 II.I~.IIIIC~~.IY o f in terest l ~ a n r e l y the posi t ion, t l re vel.,city .11111 tile ill c ( . l v r a t i u ~ ~ fo r t h e paramete rs c a n d J , t h e angul.11 ~I~,~III(~II t ~ ~ ~ t l 11,s a l ~ g u l a r ve loc i t y fo r t h e paramete r 0 a n d orrly t l ~ r I B ~ U ~ I I ~ , ~ I B 1481 (111. p;u.all~eters leng th L and n ~ a g e i t u d e u f tlac g l a a l ~ e ~ ~ t (: .I~>IIII~(,~ I,, IJC coustant . T h e K a l n ~ a l r l 1 1 1 ~ r c q l ~ , i t i t v l l ~ l.i<.tl 111 11ais paper involve discrete t i m e steps: S t . ~ t r vr, tot II*,I~I ,II ia 11set1 such t h a t XT = (zt, i t , il) i s t h e p u i t i o l ~ , v t : l~c i ~ y , a114 . r (celerat ion o f the parameter considered ( c,J,b,l.,ur (:) a t ~11c L'I' t i ~ t ~ e st p. T h e K a l m a n f i l ter ca l l I,e v iewed ct~~la i . t . i~tg 111e fu l low ing steps: T h e 111odt.1 o f t h e systr111 ~ l y l l a ~ ~ ~ i c s is: T h e e r ru r o f tile IIIVJCI ~I.IIIII rea l i t y i s given by tV,, a aerut l r ra l l w h i t e (:..ussi:u~ process uf covariance Ql : I is a t l lacr ix w l l i c l l evulves the pos i t i on x, t l ~ e veloci ty i a n d tile : ~ c c v l e r a t i u l ~ 3 f r o l ~ r one t inre s i u ~ ~ l ~ l e t o i t~ lu t l te r . 11. is easy t t i ver i fy LllaL aasun~ ing a n ~ o t i v ~ ~ wit11 r~~11st;u11 arceleratic111 le.rds 1,) 1 1 ~ f o l l o w i ~ ~ g t r ratr ix : , I I ltis is t l ~ r wv l l ~IIIBWII equa t ion o f a d r o p p t ~ l ul,jt.l I fur a ti111e IIII~ # \ d l , I t . ' l l ~ e I I I ~ ~ I Y U ~ ~ I I ~ C I I ~ 111t1de1 11sed is: ( V ) I ; ( v ) I: , 111 < * I I ~ ,111pIic g ~ ~ i c , ~ ~ , the 111easureIlhe11t ~ < 1 e l Z assu111es t11at t 11,. I I O S ~ L I S ~ I I r (1.r the value of c , J, 0 , L or (;) is Ineasur.11,1tt 11.0111 t11e l l ~ a t c l ~ i ~ ~ g 1)rocrss wllile the velocity i a11cl 111,. ; ~c . t r l r r a t l o~~ I are sot . Therefore $ is t l ~ r sc dlnr rorreS I I I ~ I I ~ I I I I ~ 111t11e position z and H1 is a in~ l~ ly tile u ~ l c e r t a i ~ ~ t y x . C'l~ousit~g this u ~ ~ c e r t a i ~ ~ t y it1 a ~ l ~ a ~ ~ n e r rt!Hectil~g our ;I lwia~ri e s t i l ~ ~ a t e of the a ~ l ~ o u n t of noise to be expectecl f r u ~ l ~ t l ~ e p~evious step ( Digitizing effects, edge ~ l e t c c t i o ~ ~ .111tI 1~11lygo11a1 a p p r u x i ~ t ~ a t i v ~ ~ ) lends to deal wit11 a s111al1 u ~ ~ c . c ~ l a i r ~ l y lor tlre p a r a ~ ~ ~ e t e r s c , 0 a r ~ d C', and a large UII cert;ri~~ly fur t l ~ e u~~reliaLle paralrleters L rlrd J due to t l ~ e I ~ I I I ~ ~ U I I I etlr< ts wllic11 break liue segnleuts. Aftt.r t l ~ e Illrasurelnent at time 1-1 has been done, t h ~ s tiwe ~ ~ l ~ ( l ; r l r ~.cluatiola predicts the aystenl state nt t i n ~ e t fro111 tllr e ? s ~ i l ~ ~ , ~ t e t l vilfues of the system state a t tilne 1-1. 111 our . ~ l ~ l ~ l i c ; ~ t i o ~ ~ , tll s equation predicts the value for z at time t I I I ~ I I I t l ~ r i ~ ~ l ' u r ~ ~ ~ a t i o n s vailable at t i ~ n e t I. ( 'ov;rri ,~~~ce pr diction for s ta te vector : 'I'llis equatio~r gives the statistics relatil~g the estur~ated JI ,rte vectors to the u ~ ~ ~ ~ ~ e a s u r a b l e ideal state vectors. ('a,varia~~ce prediction for the ~ ~ ~ e c r r i u r e ~ ~ ~ e l ~ t vector: , , 1 l ~ i r eclt1ati4111 gives the s~a tL t i c s of the e s t i~ l~a ted n~odel I~~t*.rsurrlllel~t. III our apldicatia~l~, it t l t . t e r ~ ~ ~ i ~ ~ e s the search a r t t ~ I U I L I I C 111;1tchi11g 1)rucess. '1'111s r q u a t i u ~ ~ i ~d ica t e s how 111ucl1 to weight each new 111easllrei1leltt. Note that a s111all u ~ ~ c e r t a i ~ ~ t y Rt ( precise rlleasurelllellt ) causes a large weigl~til~g Kt and therefore leads L ~ I a rurrec~ed s ta te estimate doternlined n~auily by the llleasurrtllellt. A l u g e uncertai~rty Rt causes a s~na l l weigllli~lg l i t . x t / t = x t l 1 1 t Kt(-% ~ t x t / t I ) , . I his e c l ~ ~ , i t i o ~ ~ ia used to update the s t ~ t e 111odc1. . > 1111s eqt1atit111 is used to upclate tile J L J I ~ S L ~ C S couplil~g llle rdt1111alec1 sl.klc vectors t o Ll~e u ~ ~ ~ ~ ~ e a s u r a l > l e ideal state vecL I , ~ IZor eac 11 tokel~ of the illrage 111ode1, a se l rc t i t ,~~ ( s f t l ~ e ~ .egi t~u wllere tile I I I . I ~ < ~ I ~ I I ~ process l ~ a s to seek Lm ;I ~),ossil,k I I I . I I I ~ I I I ~ ~ W C ~ > I I t,s~ke~ls is ~)rovitled tl~rougli the use .I : I I I I I ~ . I I I I ) I I I I I I< I IUI I . 'l'11W ~ I I I I ~ ~ . ~ ~ . I I t~ses l11e ~ ~ n c e r t a i ~ ~ t i e s I I ~ I I V I I ~ ~ , ~ ry 1111. l i . l l ~ ~ l , u ~ hltcril~g step OII eacl~ estinlate. It allows tu krcnl) cr111y t11v I ~ I ~ ~ . I I S withill the de;trcI~ artla. A score for err11 c . ~ , r ~ r s l ) c > ~ ~ ~ l t . ~ ~ c c is L I I I I t .~lclrlated ill urdur to d l a l ~ ~ b i g l ~ a t e J O I I I ~ ~ ~~ , s s i l , l e I I I I I I I ~ ~ ~ I ~ . 1111t01es, usillg a differe~~ce ~ ~ ~ e a s u r e . To this end, we use a 11v1111.rlhrc1 liatalrce betweell tile tokeus, detiued 3ii a weiKllted J U I I I *,I glilfrre~~ct:~ between the respective parall~eters values. OIM e .I t8,kr11 froll~ t l ~ e current frame has h e m 111atc11eiug the result of the 1ial111a11 hlteri~lg. I;or eacl~ token of tllc i ~ ~ ~ a g e Huw r ~ ~ u d e l , rel~rest.lltt.d by a fv.~lure vector of 5 CUII I I J~~I ICI ILY, we wish LO knuw whit 11 t ~ , k c i ~ 111ig11t c . ~ r r e s p o ~ ~ d to it. ' I ' l ~ i~ is Jib~le t l ~ r ~ u g l r the use ul a 3 ~ ~ ~ ~ p 1 c L V I c,f , ~ ~ ~ r i I , u t e teals I I ~ ~ I I ~ ; tlte cutrenl v.Llues uf tile Ilrra..urc, L I I C ex l~ t .< t e~ I value of tllr C U I I I ~ ~ I I I C I I L and its u l ~ c e r t . ~ i ~ ~ t y . 'l'lnii 1t*,1~1~ I, , c~ lcu la t e the M , I I I . L I ~ I I ~ L ~ ~ db ta l ce , explai~lecl ~ , ~ l u w , 1.1r e.,, 11 ci,rl~pullrnts and to ,lee Idre a toke11 of the new frc~1l~e i t ~ ~ i t l c a st..r~c 1, . I I ~ A if ill1 t l ~ e distailces ;u.e less t l ~ a n ;I fixed tl11es11(,1,1. T l ~ e co r rc s l )ond~~~ce u col~trolleIII.C ~rtwt,c.~~ the 11e.w token and tile eslill~atcd tokel~ by the I I I I ~ ~ I ~ . ~ I I I ~ ~ 01 llle es t i~l~;r t rd token. It is deliued w follows: Let t:x11 uew tbkelr, u.11et1 ftu111 tile l l ~ a t c l ~ i l ~ g process, 11e rrprese~lted by a feature vector 111 N < ~ ~ I I ~ ~ I I ~ I I L J de~roted T,,, ;rut1 ~.lre estill~ated toke11 rel,rrar~~ted Iny T,. wit11 all ~ i i ~ c e r t a i ~ ~ t y A . 'l'llr M v l ~ a l a ~ ~ o l ~ i s C ~ I I I I . I I ~ J can a p p e u or disappeiu. A new la11t.l is affected as W U V I I as a lrew scglrrellt appears a ~ r d llre prucess c u ~ ~ l . i ~ ~ u e s without .oII t . t I . I I I ~ 1 1 e t r ack i~~g a l g o r i ~ l ~ ~ n . A label co r~espo~~a l i r~g tu a good I $ , I ~ . ~ . a l ~ u ~ ~ ~ l e ~ r c e . e r ~ a i ~ r s [ l u r i ~ ~ g all the process while false corres l ~ ~ ~ ~ ~ l t ~ ~ ~ r e s cue re~noved after three frarrlos ge~~eral ly . It should I N ~o t ) i~ r~ed uut that the a l g o ~ i t l r ~ ~ ~ < a11 CIJIJC wit11 li~rr s eg~~ ien t s I I I I I V I I I ~ wit11 clillererrts ~ r r o ~ i o ~ ~ s 'I'llia i l l~~s t r a t e s the etticie~~cy of I 11.. usc.(l approach. I 5 TIIE MOTION ESTIMATION , , 1111: ~~ro l , l en~ is now to use the tokt:~rs r~rat<l~etl ovel. the sequence of i ~ ~ ~ a g e s to compute the 3D motiul~ a11,I :%I) S I ~ I I ~ I lrre ol the scerre. I I I our case the toket~s beirrg ~ ~ ~ a t c l r e ( l arr the r r t r r r r r r f~e~ of segrnerrl~. Tlre fact that they do not co r r e s l )g~~~J in general to the sanre physical point is a new n i ~ j o r source uf error. Alru~.ller L the IJreaellce of irrcorrecl mulches. Mi~rry teclr~rit~aea have bee11 proposed tu solve tlrL 1r1uLle111 wit11 li~lear 14,71 or iterative 131 a lgu r i t l~~~rs . U~r fo r t e~~a te ly they rrac11 their I ~ I I I ~ L very quickly as the noise UI the d;ttIr IIIcrrclses 12,5). Tlre iterative alg(1ri111111 we use(1, whi~11 pruve to I J ~ robust 011 real images, is l ~ a s e ~ l 011 ;I very si~rrplr collcrpt: the itIra is to look, ~ I I I O I I ~ :ill the possible nrotior~s, I.he r~lre tl1.11 ~ ~ ~ i ~ r i ~ r r i r e s the difft:rence 1,ctwet.n t l ~ r measured, a r t r~a l i ~ r ~ a g e .11111 the image obtainetl, sy~rllresizt!d using I Ibis 11101ion. *. 1. 11. 1. ., , 1 8 1,; I; ; KJ., , l-ij.-7i q i j 1, 6 Problem Definition ji: .-I.